“Hey Dan, could you deploy the coolPics branch to test? Sorry for the bother.”
“No problem, man. Tell me the SHA and I’ll deploy it.”
I had been having this conversation 4-5 times a day for a couple of weeks now. Being a huge fan of continuous integration, I wondered how to automate this. Why shouldn’t the developers be able to push whatever they wanted to test?
A colleague, overreading this back-and-forth in our HipChat room, told me to take a look at Hubot. It was custom made for automating rote tasks like webapp deployments. One weekend later, I was hooked. Here’s how I took the first step in transforming our abstract sense of team trust to tangible ownership.
Fabric deployment from an ssh login
We build, package and deploy our application via a very handy fab deployment script. This is great for parallel deploys and even includes the option of disabling hosts from our loadbalancer. However, as it runs in a sysadmin’s ssh terminal, the only feedback you get about the deployment is by asking “Is the deployment done?” or they tell you “Hey, it deployed”.
And, of course, since these deployments go through pretty quickly (5-10m), we don’t tend to wrap them in a screen command – so when the office DSL drops in the middle of a deploy…what the hell is the status now? Did we really just take a server offline and goto lunch?!
Hubot wrapper to fab and hipchat notifications
That fabric script works 99.8% of the time. We just need to harden it against network failures and make the current deployment status visible to everyone. As for actual deploys, the idea was to create 2 tiers of deployers – sysadmins and developers. If you are a developer, you can deploy to build and test environments. Only sysadmins are able to deploy live.
Since we’ve added quite a few hooks to HipChat (including JIRA issues, github commits and Jenkins build status), it was a no-brainer to make our Hubot server listen in to our room.
Jenkins test suites and GitHub Releases
Whoa – there’s a lot of stuff going on up there! sysadmin is the name of our hubot and he dutifully picks up the deploy request linking to the current deploy progress. In just this first step, we’ve made the deployment process transparent to everyone inside the hipchat room!
As fabric iterates over the servers, I’ve asked him to give a short update to the room as well to enhance the situation awareness. Since this is a test deployment, Hubot nudges Jenkins to do a minimal build and reports the results back to the room. What you don’t see here is that Jenkins records the status of these builds against the actual git SHA using github releases.
Automated Quality Gates and the Road to Continuous Deployment
Now that we can verify passing integration tests of a SHA, we can ship with confidence. If a live deployment is requested that doesn’t meet our standards of quality, Hubot can politely decline with justification. In the sometimes hectic life of a webapp developer, positive feedback loops help the team make split-second decisions that impact the company’s bottom-line.
Big time kudos to Wes Childs and his post “Hubot, Hipchat and Fabric” which obviously provided much of the technical basis for everything I’ve presented here!
Agile Web Operations 